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ABSTRACT 

A suiaary and analysis of the significant literature 
on evaluation of vocational education, focusing on a liaited nuaber 
of evaluation efforts that reflect strategies^ typically used in 
vocational education, is presented in this inforaation analysis 
paper. The paper is intended for, curriculua developers, teachers, and 
adainistrators in vocational education, state ^education departaent 
staff, and researchers. The follouing topics are discussed: purpose 
of evaluation, evaluation defined, evaluation acdels, needs \ 
assessaeht, foraer student folloii^up (including definition, 
objectives, groups to involve in conducting follcv*up studies, -^groups 
to follow-up, techniques, instruaent developaent, and conducting the 
follotf^up) , eaployer surveys, student/parent surveys, cost benefit / 
studies, inforaation systeas, state advisory council studies, and 
standards and reviews* Recoaaendations based on a reviea c£ aost of 
the aaterials included in the references list on vocational education 
progra a evaluation include the following: prograa evaluation in 
vocational education needs (1) to be a continuous effort, (2) a acre 
systeaatic approach, (3) aore eaphasis bn cost analysis, (4) 
ajSfsessaent of personnel developaent, (5) developaent of specific and 
aeasurable prograa objectives, (6) research in all areas of 
aethodology, (7) aore people involved ^ (8) systeaatic and 
coapreheusive inforaation systeas for collecting inforaation, and (9) 
eaphasis on both theoretical and practical evaluation bases* (TA) 
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FOREWORD 

The Educational Resources Information Center for Career Education 
' (ERIC/CE) is one, of sixteen clearinghouses^ in a nationwide infor- 
mation system that is funded by the National Institute of Educa- 
tion. The scope of work for ERIC/CE includes the fields of adult- 
continuing, career, and vocational-technical education. One of the 
functions of the Clearinghouse is to interpret the literature that 
is related to each of. these fields. This paper on program evalua- 
tion in vocational "education should be of particular interest to 
adult educators, business/industry personnel managers, and 
middle-aged Americans. 

The profession is indebted to Floyd McKinney, Texas ASM University 
§ Institution for his scholarship in the preparation of this paper. 
Recognition. is also due John A. Klit, Illinois State Department of 
Education, and K^y Adams, The Center for Vocational Education, The 
Ohio State UniveysJity, for their critical review of the manuscript 
prior to its fin^il revision and publication. Wesley Budke, Voca- 
tional Technical! Specialist at the ERIC Clearinghouse on Career 
Education supervjised the publication's development. Madelon 
Plaisted and Jo-Ann Cherry coordinated the production of the paper 
for publication.! 



Robert E. Taylor 

Executive Director 

The Center for Vocational Education 
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ABSTRACT 



A summary and analysis of the significant literature on .evaluation 
of vocational education, focusing .on a limited number o£, evalua- 
tion efforts that reflect strategies typically used in vocational 
education/ is presented in this information analysis, paper. The 
paper is intended for curriculum developers, teachers, and adminis- 
trators in vocational education; State education department staff; 
and researchers. The following topics are discussed: Purpose of 
evaluation, evaluation defined, evaluation models, needs assessment 
former- student followup (including definition, objectives, groups 
to involve in conducting followup studies, groups to followup, 
techniques, instrument development, and conducting the; followup) , 
en5)loyer surveys, student/parent surveys, cost/benefit studies, in- 
formation systems. State advisory council' studies, and stanijards 
and reviews. Recommendations based on a review of most of tKe ma- 
terials included in the references list on vocational education 
program evaluation include the following: Program evaluation in vo 
cational education needs (1) tc be a continuous effort, (2) a more 
systematic approach, (3) more emphasis on cost analysis, (4) assess 
meiit of personnel, development, (5) development of specific and 
measurable program objectives, (6) research in all areas of method- 
ology, (7) more people involved, (8) systematic ^nd comprehensive 
information systems for collecting information, and (9) emphasis on 
both theoretical and practical ^evaluation bases. (TA) 

Descriptors: 

*Vocational Education; Curriculam Development; Educational Adminis- 
tration Evaluation; *Educational Assessment; *Program Evaluation; 
Cost Effectiveness; Program Costs; *Program Validation; Summative 
Evaluation; *Evaluation Methods Surveys; Needs Assessment; Voca- 
tional Followup; Research Needs; Literature Reviews; Models 
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INTIIGDUCTION 



-Demand for high quality vocational education is increasing daily. 
Persons involved in these programs must haVe valid and reliable 
information, both to assess the quality of current programs, and' 
to plan the additional programs needed. / 

The need for evaluation of vocational ediication programs was 
emphasized in the Report of, the President's Panel of Consultants 
on Vocational Education in 6)63 and in the Vocational Education 
Amendments of 1968. Public Law 94-482, /Education Amendments of 
1976, calls" for federal and state evaluations of all programs 
which purport to teach entry level job /skills. As a result of 
.these. and other developments, students/ employers, and educators, 
as well as the general public, have befcome increasingly aware 
of the role of vocational- education iri preparing people for work. 

...Program evaluation must be an integral and continuous part of 
vocational education. Unless prograrts are properly evaluated, 
educators will have no basis for making decisions on program de- 
velopment and revision- As the public's dollar investment in 
education increases, there is a groWing demand that education be 
efficient and effective. For vocational educators, the issue is 
not whether to evaluate, but how. 

Much effort already has been spe^t in evaluating vocational edu- 
cation programs. A review of these efforts reveals that, while 
many of the strategies and te^niques are similar, there is no 
comprehensive and systematic- approach to evaluation. Wedemeyer 
(1969) notes several reasons for the lack of atte'ntion to evalua- 
tion: / 



1. 



Program developers d6 riot view evaluation as a necessary 
part of their professional activities. 

Evaluation receives lower priority than other activities. 

Evaluation receives minimal financial support. 

Program^develppers often are not adequately prepared to 
conduct evaluations. 



Evaluation is thieatening to many educators. 



At a National Seminar on Research in Evaluation of Occupatipnal 
Education at North Carolina State University (1968) , '^valuation 
was subdivided into a number of problems, including tRe following: 

1. The goals and objectives of vocational education haye not* 
• been succinctly stated or clearly^defined. *1 

2. There are few valid avid reliable instruments for assessing 
behavioral change. *1 

3. The outcomes of vocational education are diverse and 4^^^^" 
cult to measure. 5 

4. Evaluation efforts frequently are not used as information by 
. management. ^ 

The magnitude of the evaluation problem is further revealed in 
the Ball and Scarvia (1975) study dealing with current evaluation 
practices .in^adult technical education programs. Their findings 
show that most evaluations obtained data only through question- 
' naires and interviews of program participants. One fourth of the 
evaluations were unplanned and of poor quality. 

This paper is a summary, and analysis of the significant litera- . 
' ture on evaluation of vocational education. Time and resource 
did not permit review of all evaluative studies of vocational edu- 
cation. I have emphasized a limited number of evaluation efforts 
that reflect strategies typically used in vocational education. 
The references will guide those persons interested in further in- 
vestigation to additional resources. 

•PURPOSE OF EVALUATION 

Evaluation is an ongoing activity, and, as Barkley (1974) notes, 
it is here to stay as long as educators use public funds and work 
with people and people's children. 

Dunn (1975), applying a general definition developed by Stuff lebeam 
(1971), proposes that the purpose of evaluation in vocational edu- 
c-^tion is to provide information to resolve vocational decision-* 
making' problems. 'Barkley, also reflecting the viewpoint of • 
Stuff lebeam, further notes that evaluation is not to prove, but to 
improve. Barkley contends that evaluation for improvement purposes 
operates with two assumptions: (1) the intention of the evaluation 
program is as clear as the conceptual framework on which it is based 
and (2) the participants in the program are as anxious to provide 
information as the evaluators are to get it. 



EVALUATION DEFINED 

Evaluation has multiple meanings. Stevenson and Ward (1973) de- 
find evaluation as the phv^nomenon of examining data that reflect 
a situatioh and judging whether or not that situation is accepta- 
ble. Evaluation as'^defined by Cuba and Stufflebeam (1970) is the 
process of obtaining and providing useful Information for making 
^educational decisions. The Phi Delta Kappa Commission on Evalu^ 
ation (Stuff lebeam, 1971) found evaluation to be the process, of 
de|in»3ating, collecting, and providing information useful for 
judging decision alternatives. Several writers contend that 
evaluation must be continuous. As Wedemeyer (1969) notes, evalu- 
ation should not be a one-shot process. 



EVALUATION MODELS 



^Stevenson and Ward (1973) established three characteristics of 
an evaluation system that could be measured, to determine the 
worth of that .system: validity, effects, and cost. Validity 
was. defined as a measure of how accurately the data collection 
by the system reflect the true vocational education situation. 
Effects was defined as a measure of the impact of the results 
of evaluation on vocational programs. Cost of the system was 
prorated according to the intended effect of the system. 

McKinney, Mannebach, and Neel (1972) used an evaluation system 
in central Kentucky that emphasized students' attainment of be- 
havior as stated in the objectives. The evaluation system was 
a locally directed, state-assisted effort. Byrara (1971) notes 
that the advantage of a locally directed evaluation is that 
persons involved are those responsible for improvement of the 
program. McKinney and Mannebach (1973) also used student com- 
mittees to assist in the evaluation. Figure 1. is a diagram of 
the model developed by McKinney, Mannebach, and Neel. 

Denton (1973) used this system, to explain the types of information 
and methods which can be used to obta'.n information at each phase 
of the model. 
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.Figure 1. " 

As schools have moved toward implementing evaluation systems, the 
development of clearly stated objectives has received increasing 
emphasis. Mount Jacinto College (1972) has worked with measura- 
ble institutional objectives in moving toward accountability. 
McKinney and Mannebach (1971) conducted a workshop to assist par- 
ticipants in developing objectives stated in performance terms. 
A follow-up study to the workshop was carried out to determine 
participant activities rega.-ding the development of objectives. 

After spending nearly three years in formulation, Starr and 
Dieffenderfer (1972) proposed an evaluation system designed to^ 
assist vocational education management in program planning, 
accountability, and reporting responsibilities. Eroviding a 
core of evaluative data for program planning and evaluation, the 
system uses a management-by-objectives approach to program plan- 
ning, but is adaptable to other approaches as wall. 

The Fresno County Department of Education (1972) has developed a 
system for Program Evaluation at the Performance Objective Level 
I (PEAPOL). The system is designed to allow vocational teachers 
and district administrators to monitor closely student progress 
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and costs incurred in individual classroomsv The 
reports by linking* progress data to cost data at 
level of instruction. 

I 

Smith (1971) developed and tested a statewide sys 



ting vocational 
subsystems, one 
and l-ocation of 



education programs. Kie system c 
dealing with decisions about the 
programs and the other dealing wi 



about the qualiiy of existing vocational programs 

A model for eva] uating*;the distributive education 
Arizona has been developed by Arizona State Univc 
Students, teacher-coordinators, and business and 
were surveyed to determine if their needs were b€ 
existing program. The mi^del allows occupational 
the market/distribution cluster and an operations 
of cluster parameters. Included in the mojiel arc 
sequential steps: ; ^ ^ 

1. Identify, goals, purposes, and objectives of 
to be evaluated and the objectives of the eva 

2. Develop measurement criteria and design instr 
collect and measure needed data. 

5. Determine a valid sampling technique and coll 

4. Analyze data in terms of the objectives of th 

5. <»Report on findings and implic -ns. 

6. Make decisions for program modification based 
findings. 

A computerized model to evaluate local vocational 
grams in relation to the various functions of voc 
has been developed by Branch (1972). The evaluat 
the form of a computer print-out from individual 
cation programs and from each function within a p 
model offers an unbiased view of a vocational edu 
as it functions, notes, recommended courses of act 
of weakness, and provide an accurate job market^ a 
primary purpose of the model is to audit a progra 
of instruments collect data for the functions* of 
cation. Functions evaluated within the model are 
needs, (2) jt>b markets, (3) job performance, (4) 
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resources, (5) program planning, (6) program review, (7)- promotion, 
*(8) recruitment , (9) counseling, (10) instrucl^ion^, (11)" placement , 
and (12) evaluation. ' 



The Ohio State Department of Education's (1974) Program Review for 
Improvement, Development, and Expansion in Vocational Education 
and Guidance is composed of the following six components. 

1. ; Administrative review-focuses on board policy, administrative 

procedures, finance, program developinent, instruction, staff 
personnel, school -community relations, and evaluation and 
^ accountability. 

2. Process variable review- instructors use a lay advisory com- 

/* mittee to 'react to the variables of an'instrlzctional program. 

3. Product review-identifies succesSfes achieved by vocational 
education graduates. 

4. Cost analysis review-identifies instr&ctional cost of opera- 
ting an instructional program by' class and generates cost 
per-pupil ,per program and class. 

5. Availa bili ty an4 impact review- conducted periodically by the 
vocational, education planning district personnel to use local 
resources-'for determining community needs.' 

6. Acceptance and congruence review-uses instruments to examine 
student interests and attitudes and parent attitudes . 

, Bell!s evaluation model reported in Barkley (1974) focuses on per- 
formance of students and transactional evaluatipn. Transactional 
evaluation judges the perceptions people havQ of a jirograml Bell 
contends that the ultimate testing of any program lies in the, 

^perception of those who work in and around it. 

Finch and Bjorkquist (1975) suggest that context and input mea- 
sures offer potential for vocational education program evaluation 
(see figure 2). According to Finch and Bjorkquist, context eval- 
uation determines whether or not a program should be offered and, 
if so, what should its parameters be. Input evaluation helps to 
decide what resources and strategies will be used to achieve pro- 
gram goals and objectives. Process evaluation is used to deter- 
mine what effect the program has on students in school. Product 
evaluation examines the progriam's effects on former students. 
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Figure 2. 



Elson (1976) has developed a systematic' procedure for evaluating 
the total vocational education program in a local school. The 
system guidelines are divided into two major sections: (1) 
curriculum and instruction, including a thirty- two- item rating • 
form to be completed by vocational teachers, and (2) forms and 
instructions. 

The Illinois Department of Adult Vocational and Technical Educa- 
tion (1975) summarizes findings based on on-site evaluations. 
Their system features local self-study and use of review teams. 
Areas evaluated include (.1) students served, (2) occupational ^ 
programs, (3) administrative organization, (4)- personnel, (5) 
objectives, (6) evaluation, (7) resources used, and (8) guidance. 

Edsall (1973) suggests ten steps to guide local program evalua- 
tion. In order of use, they are (1) contacting the state voca- 
tional education department, (2) deciding how much to evaluate?.^ 
• (3) selecting the evaluating team, (4) deciding what to evaluate, 
' '1(5) orienting the evaluation te^, (6) providing materials for 
the evaluation team, (7) collecting and recording information, 
(8) reporting the results, (9) using the results, and (1) writing 
the follow-up report to the evaluation team. 



NEEDS ASSESSMENT 



Determination ot^ needs is vital to the evaluation effort, for 
identified needs must be reflected in the philosophy and obiec- 
tives of a program. Kaufman (1972) indicates that identifying 
needs is measuring the discrepancy between **lVhere are we now?** 
and "Ifliere are we to be?" Further, needs assessment must accu- 
rately reflect "~the-*'real world *' (p. 29) 
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Owings and Diener (1975) surveyed adults in the Tuscaloosa, 

.Alabanwh--aTea-ttr-assreTr5^opinions and attitudes on post^secondary ^ 

.educational opportunities and programs. From a random sample of 
approximately 400 individuals listed in th^ Tuscaloosa telephone 
directory, 202 individuals were mailed a twenty-seven-item ques- 
tionnaire. The questionnaires were followed up with telephone ' 
call's from interviewers, who recorded the questionnaire responses 
of the participants. Demographic data reflecting the personal 
characteristics of the respondents were also collected. 

An analysis of population and vocational education enrollment, 
teachers, and expenditures is included in a publication by the 
U.S. Office of Education (1971). The report is based On data re- ^ 
ceived from U.S. cities with populations over 250,000. Informa- X^^'' 
tion is -included from forty-five of the fifty-eight largest cities. \ 

\' 

Alvir (1976) developed a checklist to aid^ teachers in identifying 
needs and priorities of both sludents and teachers in vocational , 
education programs for deaf secondary school students. Alvir 
also designed a survey questionnaire to elicit information from 
the student that reveals learning and learning potential. 

Postsecondary education needs were assessed by Stelzer and Banthin 
(1975) in northeastern New York State. The. study was designed 
to grovide adul-f education planners with information for program 
planning and evaluation. Questionnaire respondents included 
1,055 individuals from four ^subregions . Data analysis focused on - 
(1) overall interst/potential market, (2) knowledge factor, 
(3) motivation factor, (4) background characteristics^ interested 
individuals, (5) approach-avoidance model (approach), and (6) 
approach- avoidance model (avoidance) . 

A survey of the four-county area surrounding Waco, Texas, was de- 
signed by Ferguson (197.^1 to assess education and training needs 
and interests of the local population and business community. A 
business and industry emploVee skill needs survey was developed 
to obtain data in the following categories: (1) general infor- 
mation, (2) employee information, (3) entrance requirements^, (4) 
^problems, and (5) supportive education programs. Jn another phase 
of the study, sampling^'units of approximately equal population size 
^w^re developed based on census tract data. The 'survey employed 
a multistage cluster sampling process with stratification for 
counties included in the survey area. Primary sampling units of 
approximately equal population size were developed within the 
counties based on census tract data from the 1970 census. 
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Randomization occurred at the following stages: 



1* Selection of primary sampling units within counties based on 
percentage of the total population of the four-county area 
in each county, 

2* Selee^^ion of the residence within the field and the use of 
random tables, 

3. ^Random^^lectiqn of the survey respondent when more than one 
, person qualified for inclusion in the sampling ^t the res i-^ 
^ dence, ^ 

Information was collected about the availability, costs, and eii- 
trance requirements of adult education programs, obstacles and] 
inducements to enrollment, interest in, specific program types, 
attitudes toward instruction and classroom nrltticjps, effective 
advertising methods, and attitudes toward adult\education and } 
vocational education. 

Alfaro (i974) developed a needs assessment system for occupa- ' 
tional education in two community college districts in Florida- 
The system includes (1) a statement of mission objectives, (2) per- 
formance^ requirements , ^and (3) .a mission profile representing the 
sequence of functions and a.function flow-back diagram showing " 
interrelationships between the various steps of the system. 

Young (1972) reported on the role of staffing informatioi: in vo- 
cational education planning- The report examines the concepts 
of staff supply and demand and the sources of relevant information 
and methodologies for its estimation/ considers criteria for de- 
termining vocational education priorities, and describes/^ method 
for allocating resources among vocational programs. ^ 

Adams (1977) conducted a national assessment of vocational educa- 
tion needs in large cities. The perceptions of needs by educators, 
students, ajid representatives from the working community were 
explored through an open-ended mail survey, personal interviews, 
and literature review. The information generated through these* 
methods were then synthesized into specifiq^^statements of need, 
clustered under thirty general goals in areas such as basic skills, 
collaboration with the .community, funding base, vocational .gui- 
dance, and community relations. National priorities for vocational 
education programs in large cities were then determined by' asking 
vocational education directors and central office staff meinbers 
to rate the statements as higher /medium, or lower priorities. 



These priorities were also compared in terms of city size, geo- 
graphic location, and minority concentratibn to determine if 
different types of cities have unique neeas. 



FORMER^OENt F0LL0W4IP 



With the increasing Emphasis on the outcome of the vocational edu- 
ca^tit^n^-s^stem as a gauge of the^.effectiveness of the program, there 

as b.een rnorr-foi-H)wiTi5nip.7of former students as an evaluative 
strategy. 



WHAT IS A FOLLOW-UP STUDY? 

A follpw-up study accumulates pertinent data from or about indivi- 
duals /after they have had similar or comparable experiences. 
FolloW-up impl^ies the collection of data about something that has 
alre,ady takeiV place. Former students are asked to reflect on how 
the/program in question either prepared them or failed to prepare 
the^ for th6ir future work. . / 

Whether ov not a follow-up study shoul<L.Be conducted depends on 
the objectives -of the evaluation. Once these objectives have 
yjeen determined, the kinds of ^ information needed to meet program 
objectives can be identified. Several techniques and procedures 
maX be/necessary to get the Heeded information. A former-student 
foirpw-up study may be onerOf.-4iiewseJLefcted procedures for infor- 
mation gathering. ^ 



OBJECTIVES . : 

Specifying objectives foi^the follow-up study is important because 
of the relationship of the objectives to the kinds of data the 
investigator needs to collect. Wentling and Lawson (1975) have 
suggested several objectives for a follow-up study, including the 
following: 

1. To determine career patterns of former participants of various 
programs . / / 

2. To determine immeJaiate demand for positions within the commu- 
nity. 

3. To determine the mobility of program graduates. 

4. To determine the adequacy of the educational or training pro- 
gram in preparing individuals for job entry. 
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5. To determine the adequacy of preparation for entry into ad^ 
vanced training, such as community college, industrial train- 
ing program, univcxsity» or adult education program, 

6. To determine adequacy of ancillary services, such as guidance 
counseling, and placement. 

7. To determine realistic job descriptions for positions obtained 
by former students or trainees. 

8. To emphasize the primary objective of career .educat5.on to 
staff and students. 

9. To provide information for required reports, (pp. 124-128) 

In evaluating the first year vocational education programs in 
Pennsylvania's correctional institutions, Lewis (1974) sought 
the opinions of the participants. The study' objectives were as 
follows: * ' . 

1. To determine the relationship between offenders » career goals, 
and their present vocational 'courses. 

2. To determine offenders' assessment of the value of their vo-^ 
cational course in obtaining employment. 

3. To determine how offenders yere selected ^f or various voca- 
tional courses, f ^ 

4. To determine offenders* self*^assessment of their skill level. 

5. To determine tKe relationship between offenders* self -assess- 
ment and instructors* assessment of their skill level. 

6. To (determine offenders' general atittude toward their course - 
work. ^ . * . ^ 

7. To determine, the »number of offenders enrolled in the voca- 
tional programs I " ■ 

8. To determine self-assessment on the quality and status of 
vocational programs. 

I 

GROUPS TO. INVOLVE IN, CONDUCTING 
FOLLOW gP STUDIES • ; 

I 

McKinney and Oglesby (f971) suggest the involvement of the coun- 
seling service, vocational education teachers, the school admini- 
stration, students, jthe school governing board, and the citizens 
advisory committee, i They also note that each school system is 
unique and tliat other groups may need to be involved in the effort. 



GROUPS' TO FOLLOW UP 

/ 

Wentling and Lawson (1975) suggest thar the selection of indivi- 
duals to be included in the follow-up study is inherent in the 

/ 

-iiJ8 



determination of the need and objectives for the follow-up. The 
j^''' importance of determining the groups to follow up is further ^sub- 
stantiateU-J)/ McKinney and Oglesby (1971)- Obviously, if the wrong 
people supply the data, the results will not be valid. 

McKinney and Oglesby (1971) suggest that if the picture of the 
total educational effort is to be realistic the follow-up study 
must include students who did not graduate as well as those who 
did graduate. In sonie schools, dropouts comprise a sizeable pro- ^ 
portion of the' classes. If this group is ignored, findings could 
be misleading. Still, many follow-up studies survey only graduates 
of a program. 

Sometimes it invaluable to follow up the total student body of a 
school or several schools. Grasso (1975) compared ^graduates from 
various high school curricula, interviewing "a random sample of 
graduates living within reasonable commuting distance. 

The length of time a student has been out of school affects the 
* answers given. McKinney and Oglesby (1971) suggest that only those 
out of school for at least a year or more shpuld be included in the 
study. Unfortunately, the longer former students have been out of - 
school, the more difficulj it is to separate the value of education 
from the influence of noneducational activities • Also, the longer 
students are away from school, the less valid their judgments 
about the current educational program are likely to be^, 



TECHNIQUES 

Commonly us \ techniques include, the mail survey, personal inter- 
view, and telephone interview. The mail survey is most commonly 
used. Lewi (1974), Hall (1975), Southwestern Collfege (1974), 
Elstehausen"(1973), Brockmann (1972), Fite and Grah (1972) repre- 
sent studies using the mail survey. In a study by Somers (1971), 
sampling was used for identifying graduates and dropouts. A ran- 
dom sample of nonrespondents were drawn f6r follow-up telephone 
interviews. ^ /' 

The personal interview, when done correctly, collects valuable 
data. However, this'' technique is usually expensive. Hall (1975) 
interviewed a random sample of graduates living within reasonable 
commuting distance. 
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INSTRUMENT DEVELOPMENT 

Wentling and Lav^on (1975) stress the importance of formulating 
objectives for the follow-up study. From the study objectives, 
Wentling and La^^son suggest the development of an instrument to 
collect the needed data. McKinney and Oglesby (1971) indicate that 
program objectives are the basis for the development of fojlow-up 
instrument items. Wentling and Lawson and McKinney and Oglesby 
review the many problems associated with developing instrument 
items and with instrument format. The Bockman and Felstehausen 
(1973) and Hall, (1975) studies provide sample instruments. 



CONDUCTING THE FOLLOW-UP 

Wentling and Lawson (1975), and McKinney and Og]esby (1971) provide 
considerable information on procedures to use in conducting the 
follow-up./ McKinney and Oglesby suggest the following mailing 
pattern. / 



•Ffirst mailing : 
'Second/mailing: 



''Alert'' card. 

Follow-up instrument, cover letter, and return 
envelope--stamped and addj:essed. 



'Third' mailing : Firs't thank-you (reminder card). 

•Four^^h mailing: Second follow-up instrument, second cover letter, 

/ and return envelope — stamped and addressed. 

•Fifth mailing : Second thank-you (reminder card). 

The Gran (1972) study used a mailing pattern with two-week inter- 
vals. At the end of four weeks, all nonrespondents were contacted 
t/y telephone. 

/ 

/The Somers (1971) study sampled graduates nationally. Dropouts 
and nonrespondents were also sampled randomly. Nonrespondents were 
cqntacted by telephone'. 



EMPLOYER SURVEYS 



Various methods have been used by vocational educators to secure 
feedback from employers about the adequacy of former- student prep- 
aration. Public Law 94-482 adds impetus to .obtaining employer re- 
actions, directing each state to determine "...the rent to which 
program completers and leavers are considered by tl* ir employers 
to be well-trained and prepared for employment." 
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Wentling and Lawson suggest the following objectives for an em- 
ployer survey: 

1. To assess performance of former studerfts, 

2* To determine how specific program graduates compare with 
graduates of other training programs. 

3. To elicit employer recommendations for improving the occu- 
pational program. - . 

4. To determine the recruitment practices of employing agencies. 

5. To assess the competency list of a specific course or program. 

6. To estimate supply and demand for individuals in particular 
occupations . 

7. To aid the public relations of the educational or training 
agency or institution, (pp. 166-169) 

The New York State Education oiepartment (1974) surveyed the members 
of , cbe American Society of Travel Agents through a questionnaire. 
The survey's aim was to determine whether the present secondary 
progralns in New York State provide sufficient career information 
and adequate preparation for young people to obtain employment in 
the trave>. industry. Willett and Piland (1973) surveyed employers 
identified from a follow-up of graduates. All employers were sent 
a questionnaire. A sample of these employers was randomly selected 
.for interview, but the investigators^found the interview phase 
time-consuming and difficult to carry out. > 

In an assessment of the radiologic technology program at Los Angeles 
City College, Gold (197 0 solicited opinions of local hospital ad- 
/ ministrators to determine the local program's effectiveness. Hall 
(1975) usea personal interviews of employees in a study of machipe 
tool technology and building construction graduates. 

STUdENT-PARENT SURVEYS 



Because they dVe most affected and coixemed by programs* students 
and parents are important sources of information in conducting 
evaluations. Unfortunately their opinions are frequently ignored 
when information is gathered. 
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Purrington (1972) examined the expectations and satisfactions of. 
students and their parents at the secondary and postsecondary level 
in traditional public school, in vocational centers, and in a com- 
munity college in Florida, The stud>^ group was determined by ran- 
domly selecting three of five vocatii'nal areas of the state and 
then selecting one vocational center, one traditional high school 
and one junior high school in each of the three^areas. 

The. acceptance and congruence review component of Ohio's Program 
Review f-^r Iniprovement, Development, and Expansion in Vocational 
Fducation and Guidance (1974) uses instruments to examine student 
and parent interests and attitudes.^ The instruments used are the 
Ohio Vocational Interest Survey and the Parent and Student Voca- 

♦ tional Education Srsrvisy^r The same instruments were also used' by 
McKinney and Manneback (1972) in the Central Kentucky Vocational 

^Education Evaluation Project. o 



COST-BENEFIT STUDIES 



IVhile cost-benefit studies are important for vocational education, 
-'twere is a lack of research in the area. The puT)ose of cost- 
benefit studies 4:s to determine the relatio ship between the, cost 
of a program and the benefits resulting from implementation of the 
program. Through" the ;^use of cost^-rbenefit studies, programs that 
produce the best possibte'Vesults '>for the least possible resource 
outlay can be identified. ^Some of the methodology used and pro- 
blems encountered are repofrted in this paper, 

Weiner '^nd Marson (1974) suggest the following outline of the basic 
methodology for conducting cost-benefit studies: 

I. Research type, ex post facto 

(Jn the Wieiner and Marson Stqdy, based-'^on information available 
on 1973-74 school year, supplemented by projected information) 

IJ. Develop course matrix for program under evaluation 

A . Coarse 'number 

B. Course title 

C. Instructor's name and annual salary 

1 . Name ^ 

2. Contract salary 

3. State retirement paid by district ' ^. 

4. Health insurance ^ * 

5. Life insurance 

6. Long-term disability 

7. Total contract salary 



G. 

I. 
J. 



Nuniber of students in course 
Program majors 
Course hour^^er weeTc 
Classroom;: Y v ' 
Percentage course- used for program under evaluation 
Number of course hdurs per week taught by instructor 
Facilities cost per cours^ ^ 
Facilities cost per student 

Facilities c^st per student enrolled in program under 
evaluation 



III. Determine economic cost input 



A. 


Instructor i 


B. 


Building 1 


C. 


Equipment ] 


D. 


Student 


E. 


Auxiliary 


F. 


Administrative 



IV, 



Determine economic Ibenef it input \ 

A. Increased tax M venues 1 

B. Increased eamiWs to the individual 

C. Increased produptivity--assumed jto be the same as item B 

D. Increased emploionent 



V. Develop, cost benefi : ration based on| economic cost and benefit 
figures obtained ,y j 

H ' i 

Wilson and Wikry (1971) have indicated S9me of the problems asso- 
ciated with costs. Theie problems include (1) adequate and dis- 
similar accounting systems, (2) lack of uniformity and precision* 
(3) difficulty in knowing how a projjram*^ costs ^ vary with expan- 
sion and contraction of variable Inputs, land (^) difficulty of es- 
timating costs arising from student's foregone earnings. . 



Estimating benefits that!| accrue to a student over a lifetime as a 
result of training also presents problems. Wilson and Wikry (1971) 
include the following as problems associated with estimating bene- 
fits: 



1. 



2. 



Lack of perfect foresight regarding a student's future income 
and the temporal pattern in which it will accrue. , 
Difficulty of ascribing extra income trainees may earn in later 
life solely to the training they receive as opposed to other 
characteristics, such as previous training, intelligence, and 
motivation. 
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3. Problem of comparing and aggregating income received at widely 
different times in the student *s lifetime* 

In addition, Wilson and Wikry (1971) note that the quantification 
of such factors- as opportunities for further training, intergenera- 
tional benefits, and increased personal satisfaction is arraost im- « 
possible. 

Barsby (1970) notes that measures of effectiveness selected to eval- 
uate a program are limited only by the resourcefulness of the in- 
vestigators. The following measures of program effectiveness are 
suggested by Barsby (1970). 

1. Academic improvement between pre- and post-program tests. 

2. Skill improvement between pre- and post-progi^am tests. 

3. Absolute levels of achievement in both academic and skill tests 
at end of program. _ . 

4. Percentage of persons completing program who are placed in 
occupations using skills -for-^which they had training! 

5. Percentage of^ persons completing program. 

6. Percentage of persons completing program who al:e placed. 

7. Employment/unero]>loyment experience following training (both 
absolute and relative to nonparticipants) . 

8. Opinions of program graduates as to how well the program pre- 
pared them for future employment. 

9^ Opinions of the immediate ""supervisors of program graduates on 
how well the program prepared the graduates for employment 
(both absolute and relative to nonparticipants). 
10. Extent to which program is serving those persons for whom it 
was designed. 

The problems associated with "before and after" a lysis (in which 
the participants serve as their own control groups are discussed 
by Barsby (1970). He agrees with others that it is extremely diffi- 
cult to show that improvements experienced by program participants 
in earnings and employment result from participation in the pro- 
gram and not from other causes. Understanding that the ideal con- 
trol group is available, Barsby suggests using the following groups. 

1. Those who entered the program but did not use the training. . 

2. Those who^ entered the program but dropped out before comple- 
tion. 

3. Those who applied for the program but did not show up 

4. Those who were registered with the employment service as needing 
jobs but did not apply for the program. 
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Friends and neighbors of program participants. 

Stromsdorfer (1972) contends that the use of control groups,* ade- 
quate sampling procedures, adjustment for nonresponse bias and self- 
selection, bias, and random probability samples of the population of 
interest- are absolute necessities. In his Review and Synthesis of 
Cost-EffeoHveness Studies of Vocational and leahnioal Education, 
Stromsdorfer suggests that cost-effectiveness studies should use 
both tabular and regression analysis. 

INFOHMATIOII SYSTEMS \ 

Vocational educators have longstanding concern for the development 
and use of meaningful information systems. However, there are few 
examples of operating' systems that are effective and efficient. 

To enable better planning and evaluation of vocational education 
programs at the state level. Young (1972) developed- a mathemat- 
ical fonmila for allocating resources to local education agencies 
which would reflect (i) staffing needs, (2) vocational education ' 
needs, (3) relative ability to pay, and (4) excess costs. 

Latta and Schmidt (1972) report on a general model for a statewide 
management information system for vocational and "technical educa- 
tion in Florida. The components included in the system are (1) 
student data, (2) instructor data, (3) program course data, (4) 
space facility use, (5) student placement and follow-up, and 
(6) fiscal data. For optimal results from the system, Latta and 
Schmidt note that vocational educators at all levels and leaders 
in industry must be involved.- Latta and Schmidt also, suggest that 
relevant inff>rpation for decision-making in vocational and adult 
education t^e provided only when the enrollment .system is aligned ^ 
with public^and private employment services, together with subsys- 
tems in industr)^T 

The Massachusetts State Department of Education (1974) designed a 
Management Information System for Occupational Education. The data 
system was^diesigned to collect and store basic census data {man- 
dated state and federal) for all occupational programs in 
Massachusetts Division of Occupational Education, including the 
annual federal reports. The data system related programs, en- 
rollments, and costs to job-entry skills acquired by program com- 
pleters in twenty program areas. ^ 
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The Tennessee Occupational Research and Developn 
Unit (1976) developed a model regional inforraati 
^rational- technical education, irxluding the foH 

!• Regional information system data sources. 

2. Regional inforiation coding system. 

^. Selected' data by county. 

4. Occupational demand subfiles, 

5. Program <jost projections. 

6. Sources of occupational training. 

7. Evaluation of vocational programs and of emp 
3 dary area vocational technical school gradua 

8. Employment status according to secondary and 
technical school graduates?. 

9. Facilities and equipment. 

10. Rules, regulations, and certification. • 

11. Individual and r'omposite data packs. 

Oliver (1973) developed a management information 
ning and evaluating vocational education program 
system is divided into a microsystem, primarily 
guidelines and systematic procedures at the stat 
system emphasizing the .ar.sessin^, planning, and 
vidua! vocational education programs in local sc 

Using information from the U.S. Office of Educat 
tional Education agencies, the Census Bureau, an 
ment of Labor, Lee '(1976) has reported on the st 
education and provided conclusions and interpret 

STATE ADVISORY COUNCIL STUDIES 



Each <3rf*the st^e a<ivi?pry^ councils issues annua 
tional education In addition, many of the coun 
special reports {;on various aspects of vocational 
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evaluative strategies used to obtain the information needed to* 
serve as a basis^ for the council reports is varied. 

The Kentucky State Advisory Council implemented a statewide eval- 
uation system developed by Adams and McCaslin (1976). The sys- 
tem^provides a vehicle for advisory committees to provide advice 
to vocational educators. The systemjias £o}ir stages: 

1. Orientation to the evaluation process^ ' / 

2. Investigation of vocational education programs by interviewing 
vocational educators, vocational studen^^and former students, 
and^ocal employers. ^ 

3. Interpretation of the interview information iirto an evaluation 
profile and report. which identifies the/inajor needs for im- 
provement, recommendations, and commendations. 

4. Communication of the results to vocational ^educators. 

The system provides information for three levels of decision- making: 
local, regional, and state. The system focuses on three major 
evaluation areas: relevance of content, job entry skills, and em- 
ployability skills. ' \ 

The South Carolina State Advisory Council on Vocational and Tech- 
nical Education employed ah outside consulting firm to conduct an 
evaluation effort. Questionnaires were sent to 10,000 of the 
largest employers- in the state to determine their perspective of 
vocational and technical education in South, Carolina. 

* '* 

In the Seventh Arariual Report of the Oklahoma State Advisory Council 
for Vocational-Technical Education, the Council reviewed the state's 
goals and priorities, cited examples with which people and their 
needs wei^e served, and made recommendations and commendations. The 
report also contains a digest of comments made at regional meetings 
with school administrators. 

As a, basis for its January 1975 report, the Washington State Ad- 
visory Council on Vocational Education conducted a study of voca- 
tional education success measures and related concerns in 
Washington State. The study was based on bamples of employers, 
graduates of vocational programs, current students in vocational 
programs, and local advisory committee members.* The sample included 
one supervisor or manager from each of 144 'firms, representing a 
cross-section by. geography, size, and institutional type* 

The sample of current students in vocational education was obtained 
by selectinjg thirty-eight vocational classes from a variety of 



-20- 



subj.ect areas. Designed to reflect the enrollments of the state as 
a whole, the sample pf students from the various types of institu- 
tions and from rural, suburban, and urban communities generally- 
reflected statewide enrollments in these same categories. All stu- 
dents attending class on the day designated for the survey were in- 
cluded in the sample. The advisory committee members associated 
with each' of the thirty-eight vocational classes were asked to re- 
spond to a written questionnaire. 



Procedures used: for most state advisory coj^hcfi rego^^ts are si^nil.^r 
to those used by Strong and Jarosik (1972)^ -in preparing the 1^72 
report for the Louisiana State Advisory Coimcil for Vocational 
Technical Education. Tfie>investigators used existing data"^'^ t 
pare the report. 



'and 
jpre- 



Kraft (1971) conducted a comprehensive study in preparing the 1971 
rep9rt for ^ the Florida JState Advisory Council on Vocational ai|d 
Technical pducatiem.-'> The study evaluative strategies includfed 
cost-benefit analysis and the securing of opinions of personnel "in 
business and industry about the vocational education program ^nd 
its graduates. Questionnaires and sample interviews were useii to 
collect data. I 

In a study to determine the effectiveness of advisory committ^e^s in 
-Florida, Danenburg (1975) randomly sampled members of committlees. 
Data were analyzed by cross- tabulations, indicating the relation- 
ship between participants' self-reported effectiveness rating and 
their responsiveness on committee activities, and by a frequencies 
program indicating the practices performed most often by each type 
of committee. 



STANDARDS AND REVIEWS 



STANDARDS 

. In an attempt to measure desired program characteristics, the U.S. 
Office of Education designed a measurement instrument for use by 
school boards, advisory groups, faculty, students, and the general 
public in evaluating vocational, and technical education programs. 
Desired program characteristics are presented in checklist format 
and may be arranged along a five-point rating scale to ascertain 

-the- degree of agreement with specific characterisitcs. 

The North Dakota State Board for Vocational Education (n.d.) has 
developed an instrument to assist schools and other institutions 
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to determine specific needs of their vocational programs and to 
provide. the State Board with accountability data for program de- 
velopment. The instrument is divided into twelve sections: (1) 
philosophy and objectives, (2) curriculum, (3) instructional staff, / 
(4) administration, (5) physical facilities and equipment, (6) / 
instructional materials and ^supplies, (7) guidance, (8) community / 
involvement, (9) student organizations, (10) advisory committees, / 
(11) students with special needs, and (12) occupational experi- / 
ence. Detailed criteria for each section are listed, with ratings 
to be indicated on a five-point continuum ranging from major imV 
provements needed to no iinprovements needed. 

The American Vocational Association (1971) has developed evaluative 
criteria and guidelines for standards and procedures that could be 
applied to vocational education at all levels. The guidelines, 
-criteria, and evaluation methods presented are divided into two 
major categories: institutional and program. Items within each 
category are grouped under the following topics: distinguishing 
characteristics, objectives, and structure and means. Each item 
is characterized in an initial statement, followed by guidelines 
for identifying and evaluating the characteristic. Forms of in- 
stitutional and individual self-evaluation are also included* 

The Division of Vocational Education, Arizona State Department of 
Education (1973) , designed an instrument for use by local education 
agencies in assessing vocational education programs. The instru- 
ZiOnt is designed to facilitate identification of program needs and 
is divided into five parts: (1) program planning, (2) program 
support, (3) student services, (4) instruction, and (5) evaluation. 
There are criteria and rating scales for each aspect of the pro- 
gram that is assessed. 

Meyer (1972) developed a guide for use by Texas ^school administra- 
tors in developing and operating programs of vocational education 
for handicapped students. 

REVIEWS 

Dobrovolny and Stark (1975) conducted a study to determine how the 
Illinois community colleges develop and approve vocational -technical 
education programs. The study was conducted by mail questionnaires 
and by on-site visits. 

The Comptroller General of the United States (1972) reviews programs 
for the U.S. Congress. One such review assesses the merits of vo- 
cational education programs receiving federal funds, ^d identified 
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existing prob!2ms in California, Michigan, OHiV, and Pennsylvania. 
Information was collected and reported under one of three head- 
ings: (1) vocational education not reaching all who need it, (2) 
funds targeted for the disadvantaged miss the mark, and (2) manage- 
ment information incomplete and inaccurate. 

Walsh (1974), through the Olympus Research Corporation, conducted * 
an ^assessment of vocational education programs for the handicapped 
under Part B of the 1968 Amendments to the Vocational Education 
Act. The study approach included (1) an assessment of program 
administration at the state level i (2) a project level assessment 
of vocational education for the handicapped, and (3) case study 
interviews with students, parents (or heads of households), and 
employers. The study focused on^»4nty-five states selected by 
a proportionately stratified probability saiiiple./%The project 
.sample was divided into two subsamples: (1) ^representative sam- 
ple--a total of ^seventy-four projects selected randomly in nine- 
teen states (those projects existing in individual statps), and 
(2) special sample--a purposive sample of ei^teen projects opera- 
ting in California, Washington, Idaho, Wyoming, and Kansas. The 
number of projects per state in the ninet^n "repi;esentative" 
states was based on each state's proportional contribution to total 
enrollment in the nineteen states. The case studies of partici- 
pants, their parents (or heads of households), and ejnployers were 
selected from projects in North Carolina, New Jersey, Illinois, 
Tqxas, and Ohio— all states included in the "representative" s'dmple. 
The criteria for selecting these states were (1) completeness of 
state data on projects and participants, (2) size of programs (total 
state enrollments), (3) representativeness of program types (special, 
regular, work study, nonwork study, etc.), (4) geographic location, 
and (5) availability of employers participating in work study 
and/ or pooperative education projects. To select a sample of non-t 
participating employers, .participating employers were categorized 
by size and type of industry. By matching other businesses within 
the locales where the participating employers were located by size 
and type of industry, a sample of nonparticipating emplayers was 
. selected. 

Brandon (1974) used i>roject Baseline data and other information to 
write an overview and informal appraisal of the Manpower Develop- 
ment and Training Act and its amendments. 

Numerous national, regional, and professional accrediting agencies 
use a review team format to assist them in conducting an appraisal 
of an agency or school to determine eligibility for accreditation. 
Because of the extensive number of accrediting agencies, no attempt 
is made in this paper to review their activities. 
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RECOMMENDATIONS 



2. 



6- 



7. 



The fbllowing recommendations are based on a review of most of the 
materials iTicljuded in the references and my experience in voca- 
tional education program evaluation. 

1. Program evaluation in vocational education needs to be a con- 
tinuous effort. Sporadic and *'one-shot'* evaluation efforts 
have some worth, but they soiaetimes force decision-makers . 
tci make inappropriate decisions. Quality decisipri-making de- ^ 
p^nds on a continuous flow of information. Most vocational 
.education programs are continuing; therefore, it is logical 
thUt the evaluation efforts also should be continuous. 
ThWe is a great need for a more systematic approach to pro- 
gram evaluation in vocational education. Too frequently, the 
evaluation effort includes a fol low-up :of students during one 
year, an employer survey two years later, a l^bor market study 
three-4Lears later, etc. The results of such"^ a study are al- 
ways in doubt. Decisions tend to be based on the latest study, 
which is not a composite of all findings. Evaluation needs to 
be comprehensive and continuous. 

More emphasis should be placed on cost analysis studies . With 
limited resources the dfecis ion-maker desperately needs' tliis 
type of information. However, decision-makers must realize 
that while 'cost analysis provides objective data, it leaves 
out some important subjective factors every decision-maker 
must consider. 

All institutions and agencies involved in personnel develop-. . 
ment should assess whether they are sufficiently emphasizing 
the preparation of personnel. Typically, program evaluation has 
not received sufficient emphasis at the preservice or inservice 
vocational education personnel development programs. 
Development of specific and measurable program objectives is^ 
essential in program evaluation. Institutions and agencies heed 
to devote more resources to the development of program objec- 
tives. As a significant part of program evaluation, it is im- 
portant to determine whether or not objectives are appropriate 
for the situation for which they have been developed. 
There is critical need for research in all are^s of methodology 
A review of follow-up studies reveals a lack of consistency in 
the use of acceptable procedures to "'conduct studies. A teview of 
other areas in evaluation shows similar problems. 
More people should be involved m evaluation. Staff, students, 
parents, lay citizens, administrators, and others should assist 
in planning, conducting, and appraising the vocational education 
evaluation process. 
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8, Systematic and comprehensive information "systems need to be 
developed for collecting information about vocational educa- 
tion programs* Program evaluation depends on a continuous flow 
of reliable and valid information. 

9. There is a lack of a concentrated^ systematic research and de- ' 
velopment. Both theoretical and practical bases of evaluation 
for vocational education need to be emphasized. 



/ 
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